{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "57216283",
   "metadata": {},
   "source": [
    "# Model Uncertainty prediction \n",
    "\n",
    "**Note**:\n",
    "\n",
    "This notebook extends the \"Custom DataLoader for Imbalanced dataset\" notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32a17a1a",
   "metadata": {},
   "source": [
    "* In this notebook we will use the higly imbalanced Protein Homology Dataset from [KDD cup 2004](https://www.kdd.org/kdd-cup/view/kdd-cup-2004/Data)\n",
    "\n",
    "```\n",
    "* The first element of each line is a BLOCK ID that denotes to which native sequence this example belongs. There is a unique BLOCK ID for each native sequence. BLOCK IDs are integers running from 1 to 303 (one for each native sequence, i.e. for each query). BLOCK IDs were assigned before the blocks were split into the train and test sets, so they do not run consecutively in either file.\n",
    "* The second element of each line is an EXAMPLE ID that uniquely describes the example. You will need this EXAMPLE ID and the BLOCK ID when you submit results.\n",
    "* The third element is the class of the example. Proteins that are homologous to the native sequence are denoted by 1, non-homologous proteins (i.e. decoys) by 0. Test examples have a \"?\" in this position.\n",
    "* All following elements are feature values. There are 74 feature values in each line. The features describe the match (e.g. the score of a sequence alignment) between the native protein sequence and the sequence that is tested for homology.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "835f1efa",
   "metadata": {},
   "source": [
    "## Initial imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9c4e5042",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/javierrodriguezzaurin/.pyenv/versions/3.10.15/envs/widedeep310/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from pytorch_widedeep import Trainer\n",
    "from pytorch_widedeep.preprocessing import TabPreprocessor\n",
    "from pytorch_widedeep.models import TabMlp, WideDeep\n",
    "from pytorch_widedeep.dataloaders import DataLoaderImbalanced\n",
    "from pytorch_widedeep.metrics import Accuracy, Recall, Precision, F1Score\n",
    "from pytorch_widedeep.initializers import XavierNormal\n",
    "from pytorch_widedeep.datasets import load_bio_kdd04\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "import time\n",
    "import datetime\n",
    "\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "046ea56a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EXAMPLE_ID</th>\n",
       "      <th>BLOCK_ID</th>\n",
       "      <th>target</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>...</th>\n",
       "      <th>68</th>\n",
       "      <th>69</th>\n",
       "      <th>70</th>\n",
       "      <th>71</th>\n",
       "      <th>72</th>\n",
       "      <th>73</th>\n",
       "      <th>74</th>\n",
       "      <th>75</th>\n",
       "      <th>76</th>\n",
       "      <th>77</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>279</td>\n",
       "      <td>261532</td>\n",
       "      <td>0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>32.69</td>\n",
       "      <td>0.30</td>\n",
       "      <td>2.5</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1256.8</td>\n",
       "      <td>-0.89</td>\n",
       "      <td>...</td>\n",
       "      <td>-8.0</td>\n",
       "      <td>1595.1</td>\n",
       "      <td>-1.64</td>\n",
       "      <td>2.83</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>-50.0</td>\n",
       "      <td>445.2</td>\n",
       "      <td>-0.35</td>\n",
       "      <td>0.26</td>\n",
       "      <td>0.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>279</td>\n",
       "      <td>261533</td>\n",
       "      <td>0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>33.33</td>\n",
       "      <td>0.00</td>\n",
       "      <td>16.5</td>\n",
       "      <td>9.5</td>\n",
       "      <td>608.1</td>\n",
       "      <td>0.50</td>\n",
       "      <td>...</td>\n",
       "      <td>-6.0</td>\n",
       "      <td>762.9</td>\n",
       "      <td>0.29</td>\n",
       "      <td>0.82</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>-35.0</td>\n",
       "      <td>140.3</td>\n",
       "      <td>1.16</td>\n",
       "      <td>0.39</td>\n",
       "      <td>0.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>279</td>\n",
       "      <td>261534</td>\n",
       "      <td>0</td>\n",
       "      <td>77.0</td>\n",
       "      <td>27.27</td>\n",
       "      <td>-0.91</td>\n",
       "      <td>6.0</td>\n",
       "      <td>58.5</td>\n",
       "      <td>1623.6</td>\n",
       "      <td>-1.40</td>\n",
       "      <td>...</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1491.8</td>\n",
       "      <td>0.32</td>\n",
       "      <td>-1.29</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-34.0</td>\n",
       "      <td>658.2</td>\n",
       "      <td>-0.76</td>\n",
       "      <td>0.26</td>\n",
       "      <td>0.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>279</td>\n",
       "      <td>261535</td>\n",
       "      <td>0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>27.91</td>\n",
       "      <td>-0.35</td>\n",
       "      <td>3.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>1921.6</td>\n",
       "      <td>-1.36</td>\n",
       "      <td>...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2047.7</td>\n",
       "      <td>-0.98</td>\n",
       "      <td>1.53</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-49.0</td>\n",
       "      <td>554.2</td>\n",
       "      <td>-0.83</td>\n",
       "      <td>0.39</td>\n",
       "      <td>0.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>279</td>\n",
       "      <td>261536</td>\n",
       "      <td>0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>28.00</td>\n",
       "      <td>-1.32</td>\n",
       "      <td>-9.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>464.8</td>\n",
       "      <td>0.88</td>\n",
       "      <td>...</td>\n",
       "      <td>-14.0</td>\n",
       "      <td>479.5</td>\n",
       "      <td>0.68</td>\n",
       "      <td>-0.59</td>\n",
       "      <td>2.0</td>\n",
       "      <td>-36.0</td>\n",
       "      <td>-6.9</td>\n",
       "      <td>2.02</td>\n",
       "      <td>0.14</td>\n",
       "      <td>-0.23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 77 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   EXAMPLE_ID  BLOCK_ID  target     4      5     6     7     8       9    10  \\\n",
       "0         279    261532       0  52.0  32.69  0.30   2.5  20.0  1256.8 -0.89   \n",
       "1         279    261533       0  58.0  33.33  0.00  16.5   9.5   608.1  0.50   \n",
       "2         279    261534       0  77.0  27.27 -0.91   6.0  58.5  1623.6 -1.40   \n",
       "3         279    261535       0  41.0  27.91 -0.35   3.0  46.0  1921.6 -1.36   \n",
       "4         279    261536       0  50.0  28.00 -1.32  -9.0  12.0   464.8  0.88   \n",
       "\n",
       "   ...    68      69    70    71   72    73     74    75    76    77  \n",
       "0  ...  -8.0  1595.1 -1.64  2.83 -2.0 -50.0  445.2 -0.35  0.26  0.76  \n",
       "1  ...  -6.0   762.9  0.29  0.82 -3.0 -35.0  140.3  1.16  0.39  0.73  \n",
       "2  ...   7.0  1491.8  0.32 -1.29  0.0 -34.0  658.2 -0.76  0.26  0.24  \n",
       "3  ...   6.0  2047.7 -0.98  1.53  0.0 -49.0  554.2 -0.83  0.39  0.73  \n",
       "4  ... -14.0   479.5  0.68 -0.59  2.0 -36.0   -6.9  2.02  0.14 -0.23  \n",
       "\n",
       "[5 rows x 77 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = load_bio_kdd04(as_frame=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "38cdc663",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop columns we won't need in this example\n",
    "df.drop(columns=[\"EXAMPLE_ID\", \"BLOCK_ID\"], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "039147fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train, df_valid = train_test_split(\n",
    "    df, test_size=0.2, stratify=df[\"target\"], random_state=1\n",
    ")\n",
    "df_valid, df_test = train_test_split(\n",
    "    df_valid, test_size=0.5, stratify=df_valid[\"target\"], random_state=1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c37401d5",
   "metadata": {},
   "source": [
    "## Preparing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "95358c9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "continuous_cols = df.drop(columns=[\"target\"]).columns.values.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4bf475a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# deeptabular\n",
    "tab_preprocessor = TabPreprocessor(continuous_cols=continuous_cols, scale=True)\n",
    "X_tab_train = tab_preprocessor.fit_transform(df_train)\n",
    "X_tab_valid = tab_preprocessor.transform(df_valid)\n",
    "X_tab_test = tab_preprocessor.transform(df_test)\n",
    "\n",
    "# target\n",
    "y_train = df_train[\"target\"].values\n",
    "y_valid = df_valid[\"target\"].values\n",
    "y_test = df_test[\"target\"].values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d37b78b7",
   "metadata": {},
   "source": [
    "## Define the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c8334b9b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WideDeep(\n",
       "  (deeptabular): Sequential(\n",
       "    (0): TabMlp(\n",
       "      (cont_norm): Identity()\n",
       "      (encoder): MLP(\n",
       "        (mlp): Sequential(\n",
       "          (dense_layer_0): Sequential(\n",
       "            (0): Linear(in_features=74, out_features=64, bias=True)\n",
       "            (1): ReLU(inplace=True)\n",
       "            (2): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (dense_layer_1): Sequential(\n",
       "            (0): Linear(in_features=64, out_features=32, bias=True)\n",
       "            (1): ReLU(inplace=True)\n",
       "            (2): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (1): Linear(in_features=32, out_features=1, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deeptabular = TabMlp(\n",
    "    column_idx=tab_preprocessor.column_idx,\n",
    "    continuous_cols=tab_preprocessor.continuous_cols,\n",
    "    mlp_hidden_dims=[64, 32],\n",
    ")\n",
    "model = WideDeep(deeptabular=deeptabular, pred_dim=1)\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "64561e0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer = Trainer(\n",
    "    model,\n",
    "    objective=\"binary\",\n",
    "    metrics=[Accuracy(), Precision(), F1Score(), Recall()],\n",
    "    verbose=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "edf1284f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "epoch 1: 100%|██████████| 3644/3644 [00:20<00:00, 175.39it/s, loss=0.0222, metrics={'acc': 0.9945, 'prec': 0.7565, 'f1': 0.6419, 'rec': 0.5574}]\n",
      "valid: 100%|██████████| 456/456 [00:01<00:00, 252.36it/s, loss=0.0125, metrics={'acc': 0.9969, 'prec': 0.92, 'f1': 0.8035, 'rec': 0.7132}]  \n",
      "epoch 2: 100%|██████████| 3644/3644 [00:20<00:00, 177.43it/s, loss=0.0119, metrics={'acc': 0.9968, 'prec': 0.9209, 'f1': 0.793, 'rec': 0.6962}] \n",
      "valid: 100%|██████████| 456/456 [00:01<00:00, 255.61it/s, loss=0.0121, metrics={'acc': 0.997, 'prec': 0.8972, 'f1': 0.8136, 'rec': 0.7442}] \n",
      "epoch 3: 100%|██████████| 3644/3644 [00:20<00:00, 176.07it/s, loss=0.0103, metrics={'acc': 0.9973, 'prec': 0.9312, 'f1': 0.8351, 'rec': 0.757}] \n",
      "valid: 100%|██████████| 456/456 [00:01<00:00, 259.70it/s, loss=0.0119, metrics={'acc': 0.997, 'prec': 0.8909, 'f1': 0.8201, 'rec': 0.7597}] \n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "trainer.fit(\n",
    "    X_train={\"X_tab\": X_tab_train, \"target\": y_train},\n",
    "    X_val={\"X_tab\": X_tab_valid, \"target\": y_valid},\n",
    "    n_epochs=3,\n",
    "    batch_size=32,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5bbc47ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train_loss</th>\n",
       "      <th>train_acc</th>\n",
       "      <th>train_prec</th>\n",
       "      <th>train_f1</th>\n",
       "      <th>train_rec</th>\n",
       "      <th>val_loss</th>\n",
       "      <th>val_acc</th>\n",
       "      <th>val_prec</th>\n",
       "      <th>val_f1</th>\n",
       "      <th>val_rec</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.022229</td>\n",
       "      <td>0.994468</td>\n",
       "      <td>0.756545</td>\n",
       "      <td>0.641866</td>\n",
       "      <td>0.557377</td>\n",
       "      <td>0.012473</td>\n",
       "      <td>0.996913</td>\n",
       "      <td>0.920000</td>\n",
       "      <td>0.803493</td>\n",
       "      <td>0.713178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.011912</td>\n",
       "      <td>0.996767</td>\n",
       "      <td>0.920918</td>\n",
       "      <td>0.792971</td>\n",
       "      <td>0.696239</td>\n",
       "      <td>0.012088</td>\n",
       "      <td>0.996981</td>\n",
       "      <td>0.897196</td>\n",
       "      <td>0.813559</td>\n",
       "      <td>0.744186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.010341</td>\n",
       "      <td>0.997341</td>\n",
       "      <td>0.931198</td>\n",
       "      <td>0.835106</td>\n",
       "      <td>0.756991</td>\n",
       "      <td>0.011884</td>\n",
       "      <td>0.997050</td>\n",
       "      <td>0.890909</td>\n",
       "      <td>0.820084</td>\n",
       "      <td>0.759690</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   train_loss  train_acc  train_prec  train_f1  train_rec  val_loss   val_acc  \\\n",
       "0    0.022229   0.994468    0.756545  0.641866   0.557377  0.012473  0.996913   \n",
       "1    0.011912   0.996767    0.920918  0.792971   0.696239  0.012088  0.996981   \n",
       "2    0.010341   0.997341    0.931198  0.835106   0.756991  0.011884  0.997050   \n",
       "\n",
       "   val_prec    val_f1   val_rec  \n",
       "0  0.920000  0.803493  0.713178  \n",
       "1  0.897196  0.813559  0.744186  \n",
       "2  0.890909  0.820084  0.759690  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(trainer.history)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94576d50",
   "metadata": {},
   "source": [
    "## \"Normal\" prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2fc67e9b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "predict: 100%|██████████| 456/456 [00:00<00:00, 689.36it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00     14446\n",
      "           1       0.91      0.78      0.84       130\n",
      "\n",
      "    accuracy                           1.00     14576\n",
      "   macro avg       0.95      0.89      0.92     14576\n",
      "weighted avg       1.00      1.00      1.00     14576\n",
      "\n",
      "Actual predicted values:\n",
      "(array([0, 1]), array([14465,   111]))\n"
     ]
    }
   ],
   "source": [
    "df_pred = trainer.predict(X_tab=X_tab_test)\n",
    "print(classification_report(df_test[\"target\"].to_list(), df_pred))\n",
    "print(\"Actual predicted values:\\n{}\".format(np.unique(df_pred, return_counts=True)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35ad6764",
   "metadata": {},
   "source": [
    "## Prediction using uncertainty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c7317302",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "predict_UncertaintyIter: 100%|██████████| 10/10 [00:05<00:00,  1.86it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00     14446\n",
      "           1       0.91      0.78      0.84       130\n",
      "\n",
      "    accuracy                           1.00     14576\n",
      "   macro avg       0.95      0.89      0.92     14576\n",
      "weighted avg       1.00      1.00      1.00     14576\n",
      "\n",
      "Actual predicted values:\n",
      "(array([0.]), array([14576]))\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "df_pred_unc = trainer.predict_uncertainty(X_tab=X_tab_test, uncertainty_granularity=10)\n",
    "print(classification_report(df_test[\"target\"].to_list(), df_pred))\n",
    "print(\n",
    "    \"Actual predicted values:\\n{}\".format(\n",
    "        np.unique(df_pred_unc[:, -1], return_counts=True)\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f63ca87a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9.99999821e-01, 1.77245539e-07, 0.00000000e+00],\n",
       "       [1.00000000e+00, 8.29310925e-11, 0.00000000e+00],\n",
       "       [9.99995947e-01, 4.06420531e-06, 0.00000000e+00],\n",
       "       ...,\n",
       "       [9.99999940e-01, 3.85314713e-08, 0.00000000e+00],\n",
       "       [1.00000000e+00, 2.98146707e-09, 0.00000000e+00],\n",
       "       [1.00000000e+00, 1.21332046e-12, 0.00000000e+00]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pred_unc"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "3b99005fd577fa40f3cce433b2b92303885900e634b2b5344c07c59d06c8792d"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
